Skip to content

Remove SQLite and deploy Postgres by default#1503

Merged
EItanya merged 4 commits intomainfrom
iplay88keys/drop-sqlite
Mar 16, 2026
Merged

Remove SQLite and deploy Postgres by default#1503
EItanya merged 4 commits intomainfrom
iplay88keys/drop-sqlite

Conversation

@iplay88keys
Copy link
Contributor

@iplay88keys iplay88keys commented Mar 13, 2026

Summary

Removes SQLite as a supported database backend and makes PostgreSQL the only supported option. Bundles a PostgreSQL instance (with pgvector) in the Helm chart so that helm install works out of the box with no external prerequisites.

Closes: #1502

Changes

Helm

  • Bundle PostgreSQL in helm/kagent/templates/postgresql.yaml — deployed when database.postgres.url and database.postgres.urlFile are both empty (the default)
  • Set database.postgres.url to skip the bundled instance and use an external PostgreSQL
  • Remove database.type, database.sqlite, SQLite emptyDir volume, and XDG_CACHE_HOME from the controller deployment

CI

  • Remove strategy.matrix: database: [sqlite, postgres] from test-e2e — single database, no matrix needed
  • Remove separate Postgres service container — postgres is deployed inside the Kind cluster by make helm-install

Helm values

database:
  postgres:
    # Leave empty to deploy the bundled postgres. Set to use external PostgreSQL.
    url: ""
    urlFile: ""
    vectorEnabled: true
    bundled:
      image: pgvector/pgvector:pg18-trixie
      storage: 500Mi
      database: postgres
      user: postgres
      password: kagent

Test plan

  • Go unit tests pass (go test -race -skip 'TestE2E.*' ./...)
  • Helm unit tests pass (helm unittest helm/kagent)
  • TestE2EMemoryWithAgent passes — confirms pgvector and memory table work end-to-end
  • Full E2E suite passes (make helm-install push-test-agent push-test-skill followed by e2e tests via go test -v github.com/kagent-dev/kagent/go/core/test/e2e -failfast -shuffle=on)

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Copilot AI review requested due to automatic review settings March 13, 2026 17:57
@iplay88keys iplay88keys requested a review from peterj as a code owner March 13, 2026 17:57
@github-actions github-actions bot added the enhancement-proposal Indicates that this PR is for an enhancement proposal label Mar 13, 2026
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@dimetron
Copy link
Contributor

Please consider support https://github.com/cloudnative-pg/cloudnative-pg
To have compatibility from the start and allow HA in production env.
@iplay88keys @EItanya

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
@EItanya EItanya merged commit 52327c3 into main Mar 16, 2026
22 checks passed
@EItanya EItanya deleted the iplay88keys/drop-sqlite branch March 16, 2026 15:57
@iplay88keys
Copy link
Contributor Author

Follow-up to this in: #1527

EItanya pushed a commit that referenced this pull request Mar 23, 2026
# Description
Follow-up to: #1503

> [!WARNING]
> If you are using a vector enabled DB for your external database, set
`database.postgres.vectorEnabled` to `true` as the default value has
changed to `false` in order to use the official `Postgres` image instead
of the `pgvector` image for the bundled database.
>
> If you were using the bundled PostgreSQL and want to keep the data,
read the mitigation section below. A direct upgrade will initialize a
fresh database.

This PR reworks how kagent's bundled PostgreSQL is configured and
deployed. The main goals are:

- **Security:** credentials are now stored in a Kubernetes Secret
instead of a ConfigMap
- **Reliability:** the bundled pod now has liveness/readiness probes, a
`Recreate` strategy, non-root security context, and a correctly
configured `PGDATA` path that survives restarts
- **Flexibility:** `bundled.enabled` and `url`/`urlFile` are now
independent — you can keep the bundled pod running while pointing the
controller at an external database, which makes migration easier
- **Clarity:** the bundled instance is more explicitly scoped to
dev/eval use; the image is switched to standard `postgres:18` in the
helm chart and the toggle is an explicit flag rather than an implicit
side effect of leaving `url` empty. The local make target deploys
`pgvector:pg18-trixie` for developing against a vector enabled database.

Note that this is a breaking change from the bundled Postgres added
earlier this week.

### What changed in the helm chart for the bundled image:

| | Before | After |
|---|---|---|
| Image | `pgvector/pgvector:pg18-trixie` |
`docker.io/library/postgres:18` |
| Username | `postgres` | `kagent` (hardcoded) |
| Database | `postgres` | `kagent` (hardcoded) |
| Mount path | `/var/lib/postgresql` | `/var/lib/postgresql/data` |
| PGDATA | `/var/lib/postgresql/data` (default) |
`/var/lib/postgresql/data/pgdata` (explicit) |
| Password storage | ConfigMap (plaintext) | Secret (base64) |
| Toggle | implicit (empty `url`/`urlFile`) |
`database.postgres.bundled.enabled` (default: `true`) |

Restarts and helm upgrades preserve data in the bundled Postgres
instance correctly once on the new chart.

### Mitigation

The bundled PostgreSQL is intended for local development. If you don't
need to keep existing data, just upgrade — the new chart initializes a
fresh database at a different path on the existing PVC and will not
touch the old data directory.

If you want to keep existing local data, you'll need to back up first
and restore after upgrading.

Backup (run before upgrading):

```bash
kubectl exec -n kagent deployment/kagent-postgresql -- \
  pg_dump -U postgres postgres > kagent-backup.sql
```

Restore (run after upgrading):
**Note** The helm chart by default uses the `Postgres` image without
vector support whereas the `helm install` make target overwrites the
image to use the `pgvector` image.

Restoring when overwriting the bundled image to use the `pgvector`
image:
```bash
PGPOD=$(kubectl get pod -n kagent -l app.kubernetes.io/component=database -o name | head -1)
kubectl exec -i -n kagent $PGPOD -- psql -U kagent -d kagent < kagent-backup.sql
```

### Database configuration reference

`bundled.enabled` and `url`/`urlFile` are independent controls:

- **`database.postgres.bundled.enabled`** controls whether the bundled
PostgreSQL pod and its PVC are deployed. It has no effect on which
database the controller connects to.
- **`database.postgres.url` / `database.postgres.urlFile`** control what
the controller connects to. When either is set, the controller uses it.
When both are empty, the controller connects to the bundled instance.

This means you can have the bundled pod running while the controller
points at an external database — useful for migrating data from the
bundled Postgres to an external Postgres.

**Connection precedence (controller only):** `urlFile` > `url` > bundled
connection string.

| Scenario | `bundled.enabled` | `url` / `urlFile` | Bundled pod
deployed? | Controller connects to |
|---|---|---|---|---|
| Default (dev/eval) | `true` | unset | yes | bundled |
| External DB, no bundled pod | `false` | set | no | external |
| External DB, bundled pod kept running | `true` | set | yes | external
|
| Bundled disabled, no external set | `false` | unset | no | nothing
(misconfigured) |

**urlFile** is recommended when your connection string contains
credentials — it keeps secrets out of Helm values and the Kubernetes
Deployment spec:

```yaml
database:
  postgres:
    urlFile: /var/secrets/db-url   # path inside the controller container
```

Mount the secret yourself via `controller.volumes` /
`controller.volumeMounts`:

```yaml
controller:
  volumes:
    - name: db-secret
      secret:
        secretName: my-postgres-url-secret
  volumeMounts:
    - name: db-secret
      mountPath: /var/secrets
      readOnly: true
```

**url** is suitable when credentials are already managed externally
(e.g. injected by a secrets manager at deploy time):

```yaml
database:
  postgres:
    url: "postgres://user:pass@db.example.com:5432/kagent?sslmode=require"
```

**bundled** deploys a single-replica PostgreSQL pod with a
PersistentVolumeClaim. Not suitable for production — no replication, no
backups, data is lost if the PVC is deleted:

```yaml
database:
  postgres:
    bundled:
      enabled: true          # default
      storage: 500Mi
      image:
        registry: docker.io
        repository: library
        name: postgres
        tag: "18"
```

---------

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement-proposal Indicates that this PR is for an enhancement proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Drop SQLite — PostgreSQL-only database backend

4 participants